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ABSTRACT 



This paper addresses the question of how to qualify a 
knowledge building dialogue in Web-based collaborative learning. It 
investigates the idea of qualifying the dialogue through grading requirements 
and through providing students with meta-awareness around their dialogue. The 
paper investigates and analyzes the changes in quality of student interaction 
in three different deliveries of the course that emerge in response to three 
different types of instruction on expected behavior. From this, preliminary 
conclusions are drawn on the extent to which descriptions of expected 
behavior influence the interactive process and the quality of the 
interaction. The first section provides an introduction. The second and third 
sections provide some brief information on the Web-based course producing the 
data of the analysis and an account of the knowledge building perspective and 
criteria of quality used in the analysis of Web-based dialogues. The fourth 
section addresses the communicative conditions of Web-based environments, 
forming part of the rationale behind the hypothesis. The fifth section gives 
a more detailed account of the research design and of the method used in 
analysis. The sixth section provides the basic analysis. The seventh section 
gives a reflection on the results of the analysis, and future research 
perspectives are discussed. (Contains 24 references.) (AEF) 
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Abstract: This paper addresses the question of how to qualify a knowledge building dialogue 
in web-based collaborative learning. It investigates the idea of qualifying the dialogue through 
grading requirements and through providing students with meta-awareness around their 
dialogue. We investigate and analyze the changes in quality of student interaction in three 
different deliveries of the course that emerge in response to three different types of instruction 
on expected behavior. From this we draw preliminary conclusions on the extent to which 
descriptions of expected behavior influence the interactive process and the quality , of the 
interaction. 



Introduction 

Web-based learning is assuming an increasing role in education at college and pre-college levels (Bates 1999; 
Collis 1996; Fjuk et al. 1999). It is not only situations of continuing education and lifelong learning which call for 
flexible organization of the learning process, e.g. with the purpose of bridging distances. The obvious advantages 
in terms of a more flexible, time and space independent organization of the learning process, also within higher 
education, are clear (Harasim 1999). The flexible organizational design of web-based learning, by itself, does not 
make this method superior to other methods. Of more importance is the quality and design of the learning 
process. Many flexible designs offer no possibilities for interaction and collaboration among students, but are 
rather sterile and lack a framework to stimulate collaborative knowledge building. Others do so, but have 
problems - even when interaction occurs - in qualifying a knowledge building dialogue. 

This paper deals with the question of how to qualify a knowledge building dialogue through grading 
requirements and as an effect from providing the students with meta-awareness around the functions of the 
requested comments in the dialogue. The paper suggests that grading requirements as well as providing awareness 
around the requirements related to the function of comments in the dialogue contribute to qualifying a knowledge 
building process in collaborative learning. 

In section 2 we provide some brief information of the web-based course producing the data of our analysis. 
Section 3 provides an account of our knowledge building perspective and the criteria of quality used in our 
analysis of web-based dialogues. Section 4 addresses the communicative conditions of web-based environments, 
forming part of the rational behind our hypothesis. In section 5 we give a more detailed account of our research 
design and of the method used in our analysis, while section 6 provides the basic analysis. We reflect on the 
results of the analysis in section 7 and discuss future research perspectives. 



Development of Dialogue Requirements in “Global Change” 

Global Change (GC) is a conventional science course for senior undergraduates or beginning graduate students at 
a US university. It gradually has been migrated to a web base over the last 6 years, with new features being 
added as ancillary software has become available. Learner-centered activities in place of or supplemental to 
conventional lectures have been introduced. 

The course consists of a sequence of learning modules on different global-change topics, each having 



evolved from a conventional university class time period. Each unit has a set of objectives, summary information 
on the topic, student-submitted collaborative (2-3 students) summary of class time discussion, “problems to 
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ponder” as discussion starters for the electronic dialogue, and extensive lists of web and other information on the 
learning module topic. Each unit has its own electronic dialogue for student discussion among themselves and 
with outside experts or representatives of selected groups. Electronic dialogue on individual learning unit topics 
is graded. The course is viewed by the designers as a laboratory for experimenting with a variety of pedagogical 
techniques and initiatives (Taber et al. 1997). 

The course went on the web in Spring 1995. Affiliated with the course was an opportunity to engage in 
electronic dialogue. The use of dialogue and associated instructor evaluation of student engagement in dialogue 
changed over the years as follows: 

1995: 

Electronic dialogue was available separately for each lecture topic, and 5% of the evaluation of student 
performance was based on whether or not a student participated. There were no minimum requirements on 
amount of discussion and no attempt was made to judge quality as part of the evaluation of students. 



1996: 

Same as 1995 except 15% of the grade depended on the electronic dialogue. Some weak attempt was made to 
judge quality but primary criterion was participation and not quantity or content. 

1997: 

Requirements for participation in electronic dialogue were substantially enhanced. For foil credit, the student 
was required to post at least 15 entries, which had to include responses to at least 6 other students’ comments. 
Furthermore, these entries had to elicit comments from 3 other students. They also were required to respond to 3 
questions issued by the instructor and to give their views on 3 additional ethical questions relating to global 
change. Thirty percent of evaluation of students was based on these numbers but no evaluation of quality was 
attempted. A virtual portfolio was introduced for each student to manage discussion. 

1998 and 1999: 

Same as 1997. 

2000 : 

Same requirements as 1998 with addition of the knowledge-building process (Stahl 1999) that required students 
to meet specific quality criteria in their discussion. Also, students were required to assess their own writing and 
demonstrate how they had met the criteria. Forty-two percent of the evaluation of student performance was 
based on discussion. 



Criteria of Collaborative Knowledge Building in Web-Based Learning 

To put our analysis of electronic dialogues in perspective, we briefly outline the set of criteria which we assert 
are signs of quality in a web-based collaborative knowledge building process. 

In the principles of collaborative learning, the process of learning is viewed to be a fundamentally social 
phenomenon, regardless of the varying theoretical emphasis in each single approach (Dillenbourg et al. 1995). 
Several other learning theories confirm this view, e.g. Etienne Wenger in his latest book, ’’Communities of 
Practice” (Wenger 2000). 

The process of knowledge building (KB) in collaborative learning, first explored by Harasim (1989), 
involves mutual exploration of issues, mutual examination of arguments, agreements and disagreements, mutual 
questioning of positions, dynamic interaction and weaving of ideas (Harasim 1989; Kaye 1992; Sorensen 1997). 
Mason (1993) finds this view to be in agreement with the communicative potential of the online environment, 
although she also points out the weaknesses of the online dialogue being that it quite often never reaches 
synthesis or closure (Mason 1993). 

In view of the generally recognized difficulties in fostering online student dialogue that converges (e.g., 
synthesizing) rather than diverges (noted by Mason), Stahl (1999) suggests a set of factors that characterize 
quality in the KB process: 

Brainstorming is the introducing of new ideas that relate to the topic or task and offer a perspective not 
previously considered; Articulating includes explaining complex or difficult concepts; 
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Reacting provides an alternative or amplified perspective on a concept previously introduced by a student; 

Organizing refers to assembling existing thoughts or perspectives in such a way that a new perspective emerges; 

Analysis includes comparing or contrasting previously articulated views or puts new understanding on existing 
data; 

Generalization takes comments or data already presented and extracts new information or knowledge that applies 
to a broader set of conditions. 

Implementing these learning quality criteria of collaborative KB requires a corresponding meta-functional 
pedagogy or instruction that facilitates and motivates such collaborative dialogue (see section 4). 



Reflective Dialogue Primary to Involved Dialogue 

Promoting knowledge-building dialogue within the context of collaborative learning appears to be a complex 
challenge. In searching for reasons for this, some research has focused on the design and nature of the 
collaborative activities implemented in the learning process (Collis 1997), and others on the quality and nature of 
the virtual environment and the evolvement of electronic inter-human dialogue (Scardamalia & Bereiter 1996; 
Sorensen 1993). Alternative studies have concluded the social aspect of a group process to be an essential 
motivator for collaboration in online learning (Harasim 1993; Cornell & Martin 1997). It seems to carry a rather 
high proportion of a group member’s inclination to engage in any interaction with the group at all (Moore & 
Kearsley 1996). This seems in total agreement with the widely acknowledged insight, that inclination to interact 
online is sensitive to the perception of interaction (Gunawardena 1 995). 

Considerable research has been directed towards the role and meta-communicative behavior of the 
instructor (Feenberg 1989). Recent studies along this line of thinking have described the online universe as a 
meta-communicative world. Contrary to the physical world in which involvement is viewed to be primary to 
reflection (Heidegger 1986) - the virtual universe provides a context and an "ontology” in which reflection may 
be said to be primary to involvement (Sorensen 1999). Assuming this new dialogical paradigm, and this primary 
position of reflection and meta-communication, it is very conceivable, from the perspective of collaborative 
interaction and dialogue in Web-based learning, that also the task of scaffolding learning processes that aim at 
supporting both interaction and (self)reflection, must move at a meta-level in terms of creating awareness of the 
function in a dialogue of a contributed comment. 



Research Design 

In our analysis we wanted to investigate the role and nature of the instructions in the requirements given to the 
students, stimulating them to interact. We wanted to assess whether m eta- communication in terms of providing 
meta-awareness of ’’the function of a comment” in the KB process, will improve the quality of the KB dialogue, 
according to the definition of quality as suggested by Stahl (1999). In our research design and analysis we make 
the following assumptions: 

1. Characteristics of discussion and comments that contribute to a KB process are (Stahl 1999): brainstorming, 
articulation, reaction, organization (including synthesis), analysis, and generalization. 

2. Use by students of characteristics of the KB process in written dialogue contributes to student learning (as 
characterized in section 3). 

Our hypothesis for the analysis is then, that by explaining the characteristics of a KB-process, and by grading 
(Sorensen & Takle 1999) student discussion on the basis of their reflected use of these characteristics, students 
will measurably increase their use of these characteristics. The rational behind the hypothesis is an 
acknowledgement of the need in primarily reflective virtual environments (as described in section 4) for 
communicatively providing meta-awareness in relation to expected communicative actions (as described in 
section 3). 

We have evaluated characteristics of student dialogue and its relationship to course requirements by 
assembling 10 comments from 1995, 10 from 1997, and 10 from 2000. These were drawn at random (although 
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not strictly with statistical rigor) and represent comments made under quite different criteria for evaluation of 
student performance. In drawing the comments from the respective databases, care was taken to draw comments 
from the same (or very closely related) topics in each of the three years. 

The data sets represent (a) comments with no obligations attached as far as evaluation of student 
performance was concerned (1995), (b) comments when numbers of responses and numbers of interactions 
among students were used for evaluation (1997), and (c) comments when the KB requirements were imposed. 
For 2000, we also have, for some database entries, the student self-assessment of their own writing. We also 
have assigned a numerical value from 1 to 10 representing the quality of the comment. 



Analysis of Dialogues 

For the purpose of our analysis we have categorized the analyzed comments in the online dialogues according to 
the knowledge building quality requirements presented by Stahl (1999) in section 3: 

• comment with no KB characteristics 

• question (a request for clarifying information) 

• analysis (comparing or contrasting previously articulated views) 

• articulation (the explanation of complex or difficult concepts) 

• reaction (presenting an alternative or amplified perspective on a concept previously introduced) 

• brainstorming (the introduction of new ideas that relate to the topic or task and offer a new perspective) 

• organization (the organization of existing thoughts or perspectives) 

• generalization (extracting new information or knowledge from comments already presented, that applies to a 
broader set of conditions) 

A level of quality (1-10) was subjectively assigned to each comment. The mean quality went from 4.4 to 3.2 to 
5.3 from 1995 to 1997 to 2000, respectively. In 1995 a preponderance entries were questions, whereas in 1997 
there were more reactions to other students. In 2000, more student comments were characterized as articulation. 

From these results, we draw the following tentative conclusions. In 1995, with no requirements on 
volume or quality of writing, students used the electronic dialogue for asking questions. The questions did not 
seem “forced” since their postings had essentially no bearing on how they were evaluated in the course. For 
1997, the overall quality of the postings went down. There seemed to be more “forced” responses. More 
questions were of a rhetorical nature, rather than seeming to originate out of curiosity about the subject matter. 
We attribute this to a sense of urgency on the part of students to meet the volume obligations of the evaluation 
scheme. The increase in quality from 1 997 to 2000 suggests that students responded to the demands for higher 
quality of writing by meeting the requirements of the KB process. A notable difference in the 2000 discussion as 
compared to previous years was the reduction in the “social” or . But at the same 

time, the number of comments increased. Comments were much more like individual essays and less 
spontaneous. 

Perhaps the largest difference over the change in dialogue requirements was in the length of each entry. In 
1995 the average length of the 10 entries was 88 words and in 1997 the average was 93 words. In 2000, 
however, the average soared to over 2,500 words. And the quality of the grammar of each posting increased 
markedly as well. This likely is attributable to the fact that students wrote their postings as essays in word 
processors on their own computers, subjected them to spell checking, and reviewed the grammar before cutting 
and pasting them into the course web-site. This increased quality and quantity came at the expense of 
spontaneity and “social” content. 



Discussion and Future Perspectives 

How do we evaluate the results of our analysis? Assuming that meta- awareness (Gutwin et al. 1995) of KB 
characteristics among students as well as grading of comments increase the process of collaborative learning, and 
accepting the value of the KB characteristics used in our analysis, we think it is fair to say that our experiment 
has proved itself to be relevant. Over the three trials the quality of student comments has increased, both as a 
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result of grading requirements and as an effect from the meta-awareness around the functions of the requested 
comments. 

Our sample size was quite small and therefore the results are subject to large uncertainty. Therefore, the 
extent to which the results can be generalized may be questioned. The selected comments, however, were drawn 
at random and taken out of the social context of the specific dialogue in which it appeared. Perhaps this could be 
considered a problem of the data set, since the KB requirements are concerned with the sequential development 
of online interaction. 

As pointed our, the largest difference over the change in dialogue requirements was in the length of each 
comment, and the increased quality of the grammar used. It seems that the more carefully composed essay-like 
style of comments as well as the increased quality and quantity came at the expense of spontaneity and “social” 
elements in the interaction. It is conceivable that dealing with electronic dialogues and an electronic comment in 
terms of a request for reflection on its function, is inhibiting the evolvement and practice of a spontaneous and 
dynamic dialogue. 

As such, the question could be posed, whether the strong meta-reflective requirements enforced on a 
dynamic phenomenon as human interaction (even though it is not in terms of formalization) appears problematic. 
Or perhaps whether the lack of recognition of social elements in the interaction works against an incitement to 
engage in collaborative interaction or knowledge building with others. Moreover, it may be pertinent as a basis 
for a potential conclusion around this aspect to specify, whether one is using data from an off-campus course 
(where the students have no complementary social interaction) or an on-campus course (where social interaction 
among students is part of daily life) as in the present study. 

In the data from our experiment there is some indication, that the “forced requirements” on the 
collaborative dialogue actually functioned like a “kick-off’ to engage in dialogue, and that it actually caused 
more interactivity than required to appear. But we find the data set from this study too small to conclude more 
precisely around this aspect. It is beyond any doubt, though, that - as default - to be forced to reflect on the 
function of one’s action before carrying it out is likely to decrease spontaneity, phatic behavior and high 
interactivity. Above all, we believe that the optimistic results of our analysis have demonstrated that, once again, 
the problem is a complex one. And perhaps the usual controversial points to discuss in this respect will be "how" 
to do it and "to what extent". All these are questions to be further studied and pursued in the future. 
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